Store provider credential overrides encrypted in the database - #492
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
jrhizor
force-pushed
the
jrhizor/db-secret-storage
branch
from
July 24, 2026 18:19
dc7aa86 to
9e96901
Compare
Fail loudly instead of silently unconfiguring providers. An Infisical read that succeeds but returns nothing is an outage, not an empty credential set, so the loader throws and the overlay keeps its last good values. Misread keys, undecryptable rows and vanished credentials log errors with the action to take. Take Infisical off the web request path. The cloud web app runs on Vercel serverless, where a runtime read would add an authentication round trip to every cold start and put Infisical in front of the whole site; Infisical's Vercel secret sync populates the environment at deploy time instead. Only the worker, which is long-lived and spends against these credentials, keeps the SDK loader — so the Infisical vars are no longer web startup requirements. Read one Infisical folder rather than recursing from the project root, so the machine identity needs no broader grant and two folders cannot both define OPENAI_API_KEY with no defined winner. Drop what the payload does not need: typ and v leave the JWE header, since ctx is what stops a ciphertext being re-homed onto another provider. Drop the hint column, which stored the last four characters of the longest value in a bundle — the password, for Oxylabs and DataForSEO — in plaintext beside the ciphertext, and which nothing reads. Delete the unused dataforseo client that captured credentials from the environment at module load.
The worker loads credentials before pg-boss starts, so a source that throws took the whole worker down. On an upgrade that source is a fresh query against provider_credentials, which fails if the database is briefly unreachable or not yet migrated — and the compose file only guarantees migration ordering when Postgres runs in Docker. Schedule the retry interval before the first load and let self-hosted modes carry on with their .env credentials when that load fails. Managed cloud still rethrows, since it has no environment fallback and a worker that started anyway would only take jobs it cannot run. This also fixes the web path never retrying after a failed initial load.
Only the worker reads them at runtime, but treating them as worker-only let a cloud deployment go out with a worker that had no way to reach Infisical. Make them cloud requirements again so that misconfiguration surfaces at startup. INFISICAL_SECRET_PATH and INFISICAL_SITE_URL stay optional — both have working defaults, and requiring the site URL would break any deployment relying on the US cloud default.
@infisical/sdk depends on @aws-sdk/credential-providers and @smithy/*, and Nitro traces a dynamic import into the output whether or not the branch can run. The cloud web app never loads credentials over the network, so that was megabytes of AWS SDK shipped into a Vercel serverless function that would never call it. Move the worker's source selection into its own entry point. startCredentialRefresh now takes a source and whether it is required, so the module the web app imports only ever references instanceCredentialSource. Tracing the import graph from apps/web/src/server.ts no longer reaches @infisical/sdk; from apps/worker/src/index.ts it still does.
Both only ever held their defaults, and each was another way for a cloud deployment to point the loader somewhere the credentials are not. Read the root of INFISICAL_ENVIRONMENT on Infisical's US cloud, with no way to override either. With the read already non-recursive, that fixes the folder contract in one place: one flat set of canonically named secrets at the environment root, and nothing else in the project is fetched.
Infisical reaches both cloud runtimes through secret syncs, which land the provider credentials in each platform's environment. The app needs no Infisical client of its own: getCredential already falls through to process.env, and with no ELMO_ENCRYPTION_KEY there is no store to consult, so cloud is pure environment with no deployment-mode branching at all. That removes the reason the refresh loop lived in @workspace/deployment, so it moves to @workspace/lib/secrets alongside the store it drives. Storage is keyed by the environment variable a secret overrides rather than by provider, since a provider is not the unit anyone edits — brightdata needs a login and a password, and grouping them only forced a bundle protocol on top of what is really a per-variable override.
`elmo init` generates the key and the CLI backfills it on upgrade, so every self-hosted deployment has one — validating it makes that guarantee visible instead of leaving a deployment silently unable to store credentials. The hosted modes are provisioned out of band and keep no store of their own, so the requirement is scoped to local.
A JWE names its own algorithms, so the allowlist on decrypt is what stops a row re-encrypted under a weaker one from being honoured. Nothing covered it. Also retires the provider-shaped AAD strings the crypto tests still used.
Without a key id there was no rotation path: a payload could not say which key it belonged to, so the store could not tell a rotated key from a corrupt row, and changing the key stranded every existing secret with no way back. ELMO_ENCRYPTION_KEY still encrypts, and the comma-separated ELMO_ENCRYPTION_KEY_OLD stays readable, so a rotation is a restart rather than a re-entry of every credential. A row naming a key nobody holds now reports that key instead of blaming the payload. The id is a domain-separated SHA-256 of the key, truncated — self-describing, so a key identifies itself with nothing to configure and nothing to keep in sync. Selecting on it is safe: the header is the AEAD's additional data, so a rewritten kid fails the tag rather than steering the payload somewhere it decrypts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the storage layer for overriding provider credentials from inside the app, so a later PR can add the UI. It is additive: nothing is moved off environment variables here. With no stored secrets, every deployment behaves exactly as it does today.
How it works
Providers read credentials through
getCredential(name)— a stored override if one exists, otherwiseprocess.env. Overrides live in asecretstable as compact JWE, keyed by the environment variable each one stands in for (OPENAI_API_KEY,OXYLABS_PASSWORD, …), and are refreshed into an in-memory overlay every minute.Keying by variable rather than by provider is deliberate: a provider isn't the unit anyone edits. BrightData needs a login and a password, and grouping them only forced a bundle protocol on top of what is really a per-variable override.
josewith direct key management (dir) andA256GCM, so application code never touches an IV or a tag.joseis already in the tree at this exact version viabetter-auth, so this adds no new dependency.Keys and rotation
ELMO_ENCRYPTION_KEYis generated byelmo initand backfilled on upgrade, and is now required in local mode — the mode the CLI provisions. The hosted modes are provisioned out of band and keep no store of their own, so they don't require it.Each payload records the id of the key that encrypted it: a domain-separated SHA-256 of the key, truncated. That makes a key self-describing, so there is nothing to configure and nothing to keep in sync. A rotation is then a restart rather than a re-entry of every credential — put the previous key in the comma-separated
ELMO_ENCRYPTION_KEY_OLDand it stays readable while new values are written under the new key. A row naming a key nobody holds reports that key rather than blaming the payload.Re-wrapping stored rows under the current key — which is what lets you finally drop
ELMO_ENCRYPTION_KEY_OLDwithout re-entering anything — belongs with the write path, so it lands with the UI.Cloud
Nothing cloud-specific, and no deployment-mode branching anywhere in this path. Infisical secret syncs land provider credentials in the web and worker environments;
ELMO_ENCRYPTION_KEYisn't set there, sogetCredentialreadsprocess.envand thesecretstable is never queried.Failure behavior
Environment credentials are always the fallback, so nothing here can fail a startup:
The worker awaits the first load so a stored credential counts toward
SCRAPE_TARGETSvalidation. The web app leaves it running in the background rather than delaying boot.Verification
pnpm test— 574 tests passed